Skip to main content
Version: 2.6

NQL User Manual

This document describes the NQL query language.

Glossary

TermDescription
fieldKey-value pair, a way of storing information, e.g. name=John. In special cases, it is allowed to store an empty value in the field. It is not possible to create a field with an empty/undefined key. The value of the field for a specific data type.
objectA collection of fields that form the basic unit of data storage. It has mandatory/system fields. A blank document has only mandatory/system fields.
collectionA logical container for storing objects. The collection can be empty, or it can be distributed. Provides an interface for querying, adding/deleting/modifying documents regardless of their internal organization.
input collectionA collection of objects (input objects) that is passed to the command.
output collectionA collection of objects (output objects) that is the result of executing a command.
streamData source. Generates (on demand or continuously) a sequence of objects (collection) of a specific type (origin).
commandProcesses a set of data (collections) coming from the input and passes the output result. Commands can be more than one and can form a so-called pipe: source1 | command1 | command2 | .... commandN. Each command has an input and an output of data (collections). The input of a command is a sequence of input objects (collection) which are the result (output) of the previous command or stream. The result of a command is a collection of output objects. A special type of command is a stream, which has only an output.
pipe (nql)A sequence of commands that process collections of objects results in the processing of data sets originating "on the left side" and passes the results "to the right". A pipeline is defined by an nql expression.
argNA function's Nth argument.
kwargA function's argument in format key=value.

Data types

TypDescription
booleanboolean value true or false.
numberA number of type: byte, int, long, float lub double
bytebyte
int32 bits integer number
long64 bits integer number
float32 bits floating point number
double64 bits floating point number
stringA sequence of characters (default type).
ip4IP address version 4
ip6IP version 6
ipIP version 4 or 6
macMAC address MAC
subnetSubnet address
datetimeDate and Time
timespana time (look timespan)
mapA collection of key->value pairs.
collectionA collection, array or set.
objectA type which means that a variable or function has a one of the types described above.
logical_expressionA boolean expression which returns a value of boolean type.

Notice: the operators or functions are to perform an internal conversion of types so that while creating a query there is no need to provide the NQL with the types each time. In the absence of a "logically" reasonable manner of procedure, the function/expression treats the data as String (default form).

For example:

asd + asd = asdasd

2 + 2 = 4

asd + 2 = asd2

timespan

timespan defines a point in time.

accepts:

  • a number value which is a number of milliseconds since EPOCH.

  • an expression containing the '@' special character.

timespanDescription
@nowCurrent time (UTC)
-1m@nowCurrent time minus 1 minute.
@2632913200000Set a fixed timestamp value to 2632913200000.
2632913200000Like above.
-1h@26329132000002632913200000 time minus 1 hour.
-15s@2007-12-03T10:15:30+01:0015 seconds earlier calculated from date and time with offset (2007-12-03T10:15:30+01:00).
-1m20s@2007-12-03T10:15:301 minute and 20 seconds earlier calculated from date and time (2007-12-03T10:15:30) local.

An example of collection of objects containing person's data.

[
{
"fName": "Jake",
"lName": "White",
"dep": "HR",
"ctry": "DE",
"age": 56,
"host": "www.facebook.com",
"docs": [
"IDCard",
"Passport",
"DrivingLicense"
],
"PD": 0.43859708144526346,
"ts": 1673755017972,
"balance": 9966.23
},
{
"fName": "Jack",
"lName": "Magenta",
"dep": "HR",
"ctry": "PL",
"age": 27,
"host": "pl.wikipedia.org",
"docs": [
"Passport"
],
"PD": 0.511210222043333,
"ts": 1674845669100,
"balance": 5556.47
},
{
"fName": "Harry",
"lName": "Watermelon",
"dep": "HR",
"ctry": "US",
"age": 49,
"host": "www.google.com",
"docs": [
"Passport"
],
"PD": 0.3029903276018222,
"ts": 1673763725949,
"balance": 7404.62
}
]

Streams

source (src)

The command feeds NQL with objects. It must always be located as the first element. Generates collections of objects from a given source in the form of an Object List. An object or a list of objects can have many forms, more or less structured and it depends on the type of source and its native capabilities. If it is, for example, a relational database or a csv file, then it will be a fixed structure form that may be a requirement and can be controlled. Sometimes, it is possible to use objects of a less complex form. It is assumed that the source has its own configuration, character (e.g. whether it is a stream or a database) which affects the form of the objects undergoing further processing.

Syntax

src stream="string" [startTime="timespan" endTime="timespan"]

Parameters

Parameter nameRequired/not required; Field typeDescription
streamYes; stringSource name with objects.
startTimeNo; timespanStart and end of the time range. Format: (see defining time).
endTimeNo; timespanEnd of time range. Format: (see defining time).

Example

src stream="{string}" [startTime="{time}" endTime="{time}"]

dst

The command stores the result to a new collector with the specified name. If a collector with the specified name already exists, the command will return an error. A collector created and fed in this way can be the data source for the next nql.

Syntax

dst {collectorName}

Parameters

Parameter NameRequired/not required; Field typeDescription
collectorNameYes; stringCollector name.

Example

Store the first 1000 objects from the source {test_data} to a collector named collector1.

src stream="testdata" | limit 1000 | dst "collector1"

collector (coll)

The command feeds NQL with objects taken from the collector with the given ID.

Syntax

coll {id} [bunchSize={number}]

Parameters

Parameter NameRequired/not required; Field typeDescription
idYes, stringIdentifier of the collector from which objects will be retrieved.
bunchSizeNo, intMaximum number of objects retrieved from the collector.

Example 1

1. src stream="testdata" | set fullName=concat(fName, " ", lName) | limit 1000 | dst "collector1"

2. coll "collector1" | sort fullName

The first nql adds the fullName field to the objects and stores the results in a collector with the "collector1" identifier . The second nql uses the "collector1" collector as the data source and then sorts it by the fullName field.


records

The command generates a collection of objects passed to the output.

Syntax

records {tuple0} [, ... {tupleN}] 

Parameters

Parameter NameRequired/not required; Field typeDescription
tuple0..NYes, stringThe collection of objects that will be passed to the command output.

Example 1

records {"fName":"Jake","dep":"IT"},{"fName":"Jack","dep":"ADM"},{"fName":"Harry","dep":"ADM"},{"fName":"Jacob","dep":"IT"}

Commands

sort

The command sorts the collection objects according to the value of the field. Sorting is performed for a maximum number of records specified in the "limit" parameter.

Syntax

| sort field1 [asc/desc] [,… fieldN [asc/desc]] [limit={number}] [outLimit={number}]

Parameters

Parameter NameRequired/not required; Field typeDescription
field1..NYes, stringThe field or list of fields by which the sorting will be performed.
asc / descNo, stringSort order from the lowest value (asc) or from the highest value (desc).
limitNo, intLimit of records that will be sorted (default is 100 thousand).
outLimitNo, intOutput record limit; changes the default limit to 1 million.

Example 1

src stream="testdata" | sort fName, lName asc limit=1000

project

The command allows you to select the object fields that will be passed to the output and/or renaming of the fields. Renaming does not change the names of the input object fields, it only causes the name to be remapped to the output object. For example, the field fName="John" in the output object will take the form of name="John".

Syntax

| project +{fieldName} as alias} / -{fieldName},...

Parameters

Parameter NameRequired/not required; Field typeDescription
+fieldNameNo, stringKeeps the "fieldName" field from the object and passes it on. The output object will have only the specified (+) fields.
-fieldNameNo, stringRemoves the "fieldName" field. Passes on an object with all fields except the specified field.
fieldName as aliasNo, stringPasses the field under a changed name.

Example 1

src stream="testdata" | project +fName, +lName

Input:

{"fName": "Jake","lName": "White","dep": "IT","ctry": "FR"} , ...

Result:

{"fName": "Jake","lName": "White"}, ...

Example 2

src stream="testdata" | project fName as Name 

Input:

{"**fName**": "Jake","lName": "White","dep": "IT","ctry": "FR"},...

Result:

{"**fName**": "Jake","lName": "White","dep": "IT","ctry": "FR"},... 

Example 3

src stream="testdata" | project -fName

Input:

{**"fName": "Jake"**,"lName": "White","dep": "IT","ctry": "FR"},... 

Result:

{"lName": "White","dep": "IT","ctry": "FR"},…

Example 4

Renaming fields in the object and removing the docs field.

src stream="testdata" | project -docs, fName as Name, lName as LastName, dep as Department, ctry as Country, age as Age, ts as TimeStamp, PD as ProbalilityOfDefault, balance as BalancePLN | limit 2

Result:

[
{
"LastName": "White",
"Name": "Jake",
"ProbalilityOfDefault": 0.27245159724157366,
"Country": "US",
"host": "www.linkedin.com",
"BalancePLN": 3366.13,
"Department": "HR",
"Age": 26,
"TimeStamp": 1675164610825
},
{
"LastName": "Magenta",
"Name": "Jack",
"ProbalilityOfDefault": 0.2100865387760955,
"Country": "DE",
"host": "www.facebook.com",
"BalancePLN": 103.09,
"Department": "HR",
"Age": 40,
"TimeStamp": 1675163862053
}
]

limit

The command truncates collections of output objects to a specified number of objects. It returns the {limit} of the first objects from the given source, starting from the {offset} position.

Syntax

| limit [{offset},] {limit}

Parameters

Parameter NameRequired/not required; Field typeDescription
offsetNo, intThe position from which objects will be returned.
limitYes, int or constant value "ALL"Limit of objects that will be returned in the result. The value can be a number or the ALL keyword can be used.

Example 1.

The first 10 objects.

src stream="testdata" | limit 10

Example 2.

The first 10 objects, starting from position 2.

src stream="testdata" | limit 2, 10 

Example 3.

All objects starting from position 1.

src stream="testdata" | limit 1, ALL

tail

The command returns the last n objects from the input collection.

Syntax

| tail {limit}

Parameters

Parameter NameRequired/not required; Field typeDescription
limitYes, intNumber of returned objects.

Example 1.

Retrieve the last two objects after sorting by fName (First Name) field.

src stream="testdata" | sort fName | tail 2

Result:

[
{
"lName": "Copper ",
"fName": "Olive",
"PD": 0.1309446006188828,
"ctry": "HU",
"host": "www.facebook.com",
"balance": 5651.48,
"dep": "HR",
"age": 40,
"ts": 1675163747517
},
{
"lName": "Burgundy",
"fName": "Olive",
"PD": 0.8857972263415672,
"ctry": "PL",
"host": "www.youtube.com",
"balance": 2013.32,
"dep": "IT",
"age": 47,
"ts": 1675163804504
}
]

set / add

The command adds or creates a new field in all records.

Set adds a new field if the field with the specified name does not exist or changes the value of an existing field. Add always adds a new field with the specified name. If a field with the specified name already exists in the object, the add command does nothing.

Syntax

| set fieldName={expression}[, … fieldNameN={expression}] 
| add fieldName={expression}[, … fieldNameN={expression}]

Parameters

Parameter NameRequired/not required; Field typeDescription
fieldName={expression}YesNew fields named "fieldName" having a value calculated with the "expression" expression .

Example 1.

Adding two new fields to objects.

Add the fullName = fName + " " + lName field and the agePlus10 = age + 10 field and retrieve the first two objects.

src stream="testdata" | set fullName = concat(fName, " ", lName), aggPlus10 = add(age, 10) | limit 2

Result:

[
{
"lName": "White",
"fName": "Jake",
"PD": 0.27245159724157366,
"ctry": "US",
"host": "www.linkedin.com",
"balance": 3366.13,
"dep": "HR",
"age": 26,
"ts": 1675164610825,
"fullName": "Jake White",
"agePlus10": 36
},
{
"lName": "Magenta",
"fName": "Jack",
"PD": 0.2100865387760955,
"ctry": "DE",
"host": "www.facebook.com",
"balance": 103.09,
"dep": "HR",
"age": 40,
"ts": 1675163862053,
"fullName": "Jack Magenta",
"agePlus10": 50
}
]

top / rare

The command returns the {limit} of the most frequent (top)/rarest (rare) values of the expression.

Syntax

| top {field} [by {field_name, [field_name1, ….]}] [limit=number] [others=true/false]  [maxBuckets=number]

| rare {field} [by {field_name, [field_name1, ….]}] [limit=number] [others=true/false] [maxBuckets=number]

Parameters

Parameter NameRequired/not required; Field typeDescription
fieldYes, stringThe name of the field from which values will be retrieved.
byNo, stringGrouping by fields.
limitNo, NumberNumber of returned results.
othersNo, BooleanIf set to true, the number of counts for the other values will be added to the results.
maxBucketsNo, NumberThe number of buckets returned as a result.

Example 1.

Search for the names of the two departments that are the most frequent in the objects.

src stream="testdata" | top dep limit=2

Result:

[
{
"dep": "ADM",
"_count": 390
},
{
"dep": "IT",
"_count": 350
}
]

Example 2.

Search for the names of the two departments that are the least frequent in the objects.

src stream="testdata" | rare dep limit=2

Result:

[
{
"dep": "HR",
"_count": 250
},
{
"dep": "IT",
"_count": 350
}
]

Example 3.

For each value of the fName (Name) field, retrieve the two least frequent departments.

src stream="testdata" | rare dep by fName limit=2

Result:

[
{
"fName": "Joe",
"dep": "HR",
"_count": 10
},
{
"fName": "Joe",
"dep": "IT",
"_count": 10
},
{
"fName": "Harry",
"dep": "IT",
"_count": 20
},
{
"fName": "Harry",
"dep": "HR",
"_count": 30
},
{
"fName": "James",
"dep": "HR",
"_count": 20
},
{
"fName": "James",
"dep": "IT",
"_count": 30
},
{
"fName": "Ethan",
"dep": "IT",
"_count": 10
},
{
"fName": "Ethan",
"dep": "HR",
"_count": 40
},
...

fork

The command takes objects from the input collection and passes them for parallel processing by one or more pipelines. This is how parallel processing of object collections can be implemented.

Syntax

| fork ({subPipe1}), ({subPipe2}) [, ... ({subPipeN})] 

Parameters

Parameter NameRequired/not required; Field typeDescription
subPipe1..NYes; nql expressionA pipeline for parallel execution.

Example 1. Parallel execution of two streams.

src stream="testdata" | fork (set fullName=concat(fName, " ", lName) | limit 2), (aggr fieldsCount=sum(age) by fName,lName maxBuckets=2) 

Result:

[
{
"lName": "White",
"fName": "Jake",
"PD": 0.27245159724157366,
"ctry": "US",
"host": "www.linkedin.com",
"balance": 3366.13,
"dep": "HR",
"age": 26,
"ts": 1675164610825,
"fullName": "Jake White"
},
{
"lName": "Magenta",
"fName": "Jack",
"PD": 0.2100865387760955,
"ctry": "DE",
"host": "www.facebook.com",
"balance": 103.09,
"dep": "HR",
"age": 40,
"ts": 1675163862053,
"fullName": "Jack Magenta"
},
{
"fieldsCount": 260,
"_id": [
"Jake",
"White"
]
},
{
"fieldsCount": 400,
"_id": [
"Jack",

"Magenta"
]
}
]

The result is a composite (union all) of the results from two pipelines.


subpipe

The command is used to block processing until all parallel subPipes have executed.

SubPipe rules:

  • each must contain an src command and a dst command,
  • if any of them ends in an error then the parent pipe is aborted and an error is returned.

Syntax

| subPipe ({subPipe1}), [, ... ({subPipeN})] 

Parameters

Parameter NameRequired/not required; Field typeDescription

sleep

The command is used to delay the execution of the next command in the pipeline.

Syntax

| sleep {delay}

Parameters

Parameter NameRequired/not required; Field typeDescription
delayYes; NumberDelay in milliseconds of executing the next command in the pipeline.

unwind

The command unwinds the collection values for the {field} field. A new object is created for each value. The record remains unchanged if the {field} field is not a collection.

Syntax

| unwind {field} [includeEmpty=true/false] 

Parameters

Parameter NameRequired/not required; Field typeDescription
fieldYes; stringThe name of the field from which values will be taken for unwinding. If the field is not a collection, the output object remains unchanged.
includeEmptyYes; BooleanIf true, it also unwinds empty collection items.

Example 1. Unwinding of the array in the "docs" field to two objects.

src stream="testdata" | project +fName, +lName, +docs | unwind docs

Input data:

...
{"fName": "Jake","lName": "White","docs":["IDCard","Passport"]}
...

Result:

[
...,
{"fName": "Jake","lName": "White","docs":"IDCard"},
{"fName": "Jake","lName": "White","docs","Passport"},
...
]

fields

The command displays information about fields. It counts fields, unique values and collects the most common values.

Syntax

| fields [maxFields=number] [maxTuples=number ] [topCount=number ] [maxDistinctValuesPerField=number ] 

Parameters

Parameter NameRequired/not required; Field typeDescription
maxFieldsNo; intMaximum number of fields processed by the command.
maxTuplesNo; intMaximum number of processed records (tuples).
maxDistinctValuesPerFieldNo; intThe maximum number of unique values that will be counted individually.
topCountNo; intThe number of most common values to show.

where

The command filters the input objects with the specified logical filter (logical_filter).

Syntax

| where {logical_filter}

Parameters

Parameter NameRequired/not required; Field typeDescription
logical_filterYes; logical_expressionA filter that is a Boolean expression. The expression must contain the operators described in the table below.

Operators

Logical operatorsExample
AND ($and)a=1 AND b=2
OR ($or)a=1 AND ( b=2 OR b=3)
NOT ($not)NOT a=1 AND NOT b=2
Comparison operatorsExample
= ($eq)ctry="US" lub $eq(ctry,"US")
!= ($neq)fName!="Jake"
> ($gt)age > 40 lub $gt(age, 40)
>= ($gte)age >= 40 lub $gte(age, 40)
< ($lt)age < 40 lub $lt(age, 40)
>= ($lte)age <= 40 lub $lte(age, 40)
$contains$contains(fName,"org")
$not_contains$not_contains(fName,"org")
$startsWith$contains(fName,"Ja")
$endsWith$contains(fName,"ke")
$isEmpty$isEmpty(fName)
$isNotEmpty$isNotEmpty(fName)
$isNull$isNull(fName)
$isNotNull$isNotNull(fName)
$in$in(fName,["Jake", "George"])
$notIn$notIn(fName,["Jake", "George"])
~= ($like)fName~="Ja?e*"
=/regexp/ ($regex)fName=/^Jac.

If you use the operators "=","!=",">=",">","<=" and "<", there must be a constant value or a regular expression on the right side of the operator, for example fName="John" - right, lName=/J.*n$/ - right, fName=lName - wrong, age>PD - wrong.

The AND, OR and NOT operators have a priorities like + (plus) and * (multiplication) execution in arithmetic math expressions where AND is equal to * and OR is +.

For example, we have an expression a=1 OR b=2 AND NOT c=3 OR d=4. Then first, the 'NOT c=3' will be executed and result stored in hidden valiable ret1, then the 'b=2 AND ret1' and result stored in ret2 and finally the 'a=1 OR ret2 OR d=4' expression.

Example 1

Select people with the name Jake or George and an age greater than 40 and not living in the country "US".

src stream="testdata" | where (fName="Jake" or fName="George") and age>40 and ctry!="US"

Result:

[
{
"lName": "Burgundy",
**"fName": "George",**
"PD": 0.16417805586318624,
**"ctry": "IE",**
"host": "www.youtube.com",
"balance": 2932.06,
"dep": "ADM",
**"age": 43,**
"ts": 1675164422754
},
{
"lName": "Inkjet",
**"fName": "Jake",**
"PD": 0.708725767372459,
**"ctry": "HU",**
"host": "www.google.com",
"balance": 7645.02,
"dep": "IT",
**"age": 49,**
"ts": 1675163960249
},
{
"lName": "Lavender",
**"fName": "George",**
"PD": 0.2689487262832958,
**"ctry": "IT",**
"host": "www.linkedin.com",
"balance": 6463.79,
**"dep": "IT",**
"age": 42,
"ts": 1675163920281
},
{
"lName": "Beige",
**"fName": "Jake",**
"PD": 0.8439732967760376,
**"ctry": "IT",**
"host": "www.linkedin.com",
"balance": 9882.36,
**"dep": "IT",**
"age": 42,
"ts": 1675164552379
},
...

aggr

Aggregation command. As a result of the aggregating function (accumulator) (see AggregatingFunctions) you receive a single result for the data set being a collection of objects or its subset. It is possible to use the aggregating functions in the aggr command on various fields, however it is not possible to combine aggregating functions with regular functions. For example, the src stream ="netflowByCountryAggr" | aggr mintimestamp=tsToStr(min(timestamp)) command which is supposed to retrieve the minimal (min) value of the timestamp field and immediately convert the result to text form (tsToStr), is incorrect and will not work. In this case, you can use the set command: src stream ="netflowByCountryAggr" | aggr mintimestamp=min(timestamp) | set mintimestampStr=tsToStr(mintimestamp).

Syntax

| aggr {field_name}={aggregate_function(...)} [,{field_name2}={aggregate_function_namej},...] [ by {field_name_by}, [{field_name}, ...] ] [unwind={bool}] [maxBuckets={number}] 

Parameters

Parameter NameRequired/not required; Field typeDescription
field_nameYes; stringThe name of the field whose values will be calculated by the given aggregation function.
aggregate_functionYes; functionOne of the aggregating functions described in the table with aggregating functions (AggregatingFunctions).
by field_nameNo; stringThe name of the field(s) by which the data is grouped and returned as buckets.
unwindNo; booleanIf true, the values of the aggregated fields will not be in the array, but will be unwinded to objects.
maxBucketsNo; intMaximum number of buckets for which aggregation is performed.

Example 1.

Taking the maximum and minimum age of people in each department.

src stream="testdata" | aggr maxAge=max(age), minAge=min(age) by dep 

Result:

[
{
"maxAge": 59,
"minAge": 23,
"_id": [
"HR"
]
},
{
"maxAge": 56,
"minAge": 21,
"_id": [
"ADM"
]
},
{
"maxAge": 60,
"minAge": 20,
"_id": [
"IT"
]
}
]

The value of the grouping field "dep" is returned in an array in the "_id" field.

Example 2.

Taking the maximum and minimum age of people in each department.

Unlike Example 1, here there is an unwinding to the aggregation field rather than the _id array.

src stream="testdata" | aggr maxAge=max(age), minAge=min(age) by dep unwind=true

Result:

[
{
"maxAge": 59,
"minAge": 23,
"dep" : "HR"
},
{
"maxAge": 56,
"minAge": 21,
"dep" : "ADM"
},
{
"maxAge": 60,
"minAge": 20,
"dep" : "IT"
}
]

The value of the grouping field "dep" is returned in an array in the "_id" field.

Example 3.

Retrieving the maximum and minimum age of people in each department. The data aggregation will be performed for a maximum of two buckets.

src stream="testdata" | aggr maxAge=max(age), minAge=min(age) by dep maxBuckets=2

Result:

[
{
"maxAge": 59,
"minAge": 23,
"_id": [
"HR"
]
},
{
"maxAge": 56,
"minAge": 21,
"_id": [
"ADM"
]
}
]

Example 3.

Retrieving the number of balances (valuesCount) and the value of the highest balance (maxBalance) for each person.

src stream="testdata" | aggr valuesCount=count(balance), maxBalance=max(balance) by fName, lName

Result:

[
{
"_id": [
"Jake",
"White"
],
"maxBalance": 8139.64,
"valuesCount": 10
},
{
"_id": [
"Jack",
"Magenta"
],
"maxBalance": 9841.06,
"valuesCount": 10
},
{
"_id": [
"Harry",
"Watermelon"
],
"maxBalance": 8199.47,
"valuesCount": 10
},
...

rangeAggr

Aggregating command. It returns the results for specified subsets of data defined using a range of values.

| rangeAggr field1=aggregate_function(field2) on field5 [by field3[, field4]] ranges="1-5,5-10"  [bucketAlias={string}] [unwind={bool}] 

Parameters

Parameter NameRequired/not required; Field typeDescription
field1Yes; stringName of the field to which the result will be assigned
aggregate_functionYes; functionOne of the aggregating functions, accumulators for example: count,avg,dc etc....
field2Yes; stringThe field whose values will be aggregated
on field5Yes; stringField whose values are checked for belonging to ranges
rangesYes; stringRanges defined by a string of characters with the format: {from-to},{from-to}. Note: ranges on the left are inclusive and on the right are exclusive.
by {field3}[,{field4}]No; stringValues grouping field.
bucketAliasNo; stringAlias for the "bucket" field.
uwindNo; booleanUnwind values.

Example 1.

Retrieving the maximum balance (maxBalance) and the number of samples in the given range (valuesCount) for each person. Data for aggregation will be calculated and shown for two ranges (_bucket) of the balance field values: {0,5000) and {8000,10000).

src stream="testdata" | rangeAggr valuesCount=count(balance), maxBalance=max(balance) on balance by fName,lName ranges="0-5000,8000-10000"

Result:

[
{
"valuesCount": 9,
"_bucket": "0-5000",
"maxBalance": 4683.53,
"_id": [
"Jake",
"White"
]
},
{
"valuesCount": 4,
"_bucket": "0-5000",
"maxBalance": 2981.42,
"_id": [
"Jack",
"Magenta"
]
},
{
"valuesCount": 3,
"_bucket": "0-5000",
"maxBalance": 4384.36,
"_id": [
"Harry",
"Watermelon"
]
}
...
{
"valuesCount": 3,
"_bucket": "8000-10000",
"maxBalance": 9241.08,
"_id": [
"Connor",
"Amber"
]
},
{
"valuesCount": 4,
"_bucket": "8000-10000",
"maxBalance": 9926.31,
"_id": [
"Richard",
"Carrot"
]
},
{
"valuesCount": 2,
"_bucket": "8000-10000",
"maxBalance": 9015.39,
"_id": [
"Justin",
"Orange"
]
},
{
"valuesCount": 3,
"_bucket": "8000-10000",
"maxBalance": 8940.63,
"_id": [
"Jacob",
"Turquoise"
]
}
...
]

splitAggr

The command performs several aggregations, with each subsequent aggregation being performed in the context of the previous one.

Syntax

| splitAggr field1=aggregate_function(streamField1) [, ... fieldN=aggregate_function(streamFieldN)] ({subAggr1}), ({subAggr2}), [, ... ({aubAggr3})] [unwind={boolean}] 

Parameters

Parameter NameRequired/not required; Field typeDescription
field1Yes; stringName of the field to which the result will be assigned
aggregate_functionYes; functionOne of the aggregate functions, accumulators for example: count,avg,dc etc....
streamField1Yes; stringThe field whose values will be aggregated
subAggrYes; stringSub-aggregation. A minimum of two must be defined.
uwindYes; booleanUnwound values

Example 1. Average balance for a given country and department

In this example, the calculation of values uses two pipelines made with the fork command that return the same result. The first, using the splitAggr command, initially sums (performs an aggregation) the values of the "balance" field for a given country (ctry), and then on the result obtained, sums (performs an aggregation) the values of "balance" for departments. The result is stored in the "sumBytes0" field. The second, using the aggr command, performs summing "at once" for the dimension: country + department. The result is stored in the field "sumBytes2".

src stream="testdata" 
| fork (splitAggr sumBytes0=sum(balance) (aggr by ctry unwind=true) ,(aggr by dep unwind=true) unwind=true ),
(aggr sumBytes2=sum(balance) by ctry, dep unwind=true )
| sort ctry,dep

Result:

{
"status": {
"state": "FINISHED"
},
"total": 42,
"data": [
{
"ctry": "DE",
"sumBytes0": 666112.88,
"dep": "ADM"
},
{
"ctry": "DE",
"dep": "ADM",
"sumBytes2": 666112.88
},
{
"ctry": "DE",
"sumBytes0": 127414.63000000002,
"dep": "HR"
},
{
"ctry": "DE",
"dep": "HR",
"sumBytes2": 127414.63000000002
},
{
"ctry": "DE",
"sumBytes0": 264624.96,
"dep": "IT"
},
{
"ctry": "DE",
"dep": "IT",
"sumBytes2": 264624.96
},
...
]
}

timeAggr

An command that aggregates data using an aggregating function (accumulator). Returns results for specific subsets of data defined as time periods. It is primarily used to create graphs where one of the axes corresponds to time.

Syntax

| timeAggr field1=aggregate_function(field2) [on field5] [by field3[, field4]] [interval={timespan}] [dir={dir}] [maxBuckets={number}] [bucketAlias={string}] [unwind={bool}]

Parameters

Parameter NameRequired/not required; Field typeDescription
field1Yes; stringName of the field to which the result will be assigned.
aggregate_functionYes; functionOne of the aggregate functions, accumulators for example: count,avg,dc etc...
filed2Yes; stringThe field whose values will be aggregated.
by field3 ,field4Yes; stringValue grouping field(s).
on field5Yes; stringThe field from which time is taken for counting buckets, by default it is the timestamp field.
intervalYes; stringThe timespan for which values are to be calculated, e.g. 1d means a separate value for each day.
dirNo; stringBucket sorting direction. Accepts "asc" or "desc" values. No sorting by default.
maxBucketsNo; intMaximum number of buckets (default: 10 thousand).
bucketAliasNo; stringAlias for "_bucket" field.
unwindNo; booleanIf true then unwind the values in the "_id" field.

Example 1. The average age and number of people on each day for which PD and balance data is assigned to.

src stream="testdata" | timeAggr  avgAge=avg(age), cnt=count(age) on ts interval="1d" | set dateTime=tsToStr(_bucket) | sort _bucket

Result:

[
{
"avgAge": 39.5,
"cnt": 30,
"_bucket": 1672531200000,
"dateTime": "2023-01-01 00:00:00.000"
},
{
"avgAge": 42.46153846153846,
"cnt": 26,
"_bucket": 1672617600000,
"dateTime": "2023-01-02 00:00:00.000"
},
{
"avgAge": 43.28947368421053,
"cnt": 38,
"_bucket": 1672704000000,
"dateTime": "2023-01-03 00:00:00.000"
},
{
"avgAge": 42.125,
"cnt": 32,
"_bucket": 1672790400000,
"dateTime": "2023-01-04 00:00:00.000"
},
{
"avgAge": 39.121212121212128,
"cnt": 33,
"_bucket": 1672876800000,
"dateTime": "2023-01-05 00:00:00.000"
},
{
"avgAge": 43.0,
"cnt": 36,
"_bucket": 1672963200000,
"dateTime": "2023-01-06 00:00:00.000"
},
{
"avgAge": 40.97674418604651,
"cnt": 43,
"_bucket": 1673049600000,
"dateTime": "2023-01-07 00:00:00.000"
},
...

Functions


now

The now() function is used to retrieve the timestamp in the GMT+00 time zone at the time of the call.

Syntax

now() : long

Return value

The function returns the current timestamp given in milliseconds. The type of the returned value is long. The returned value is always in the GMT+00 time zone.

Example 1

ExampleResult
now()1622653803000 (2 June 2021 17:10:03)

timestamp

The function calculates the timestamp for the given values of the year, month, day, hour, minute, and second.

Syntax

timestamp(year, month, day, hour, minute, second [, offset=int]) : long

Parameters

Parameter NameRequired/not required; Field typeDescription
yearYes; intYear
monthYes; intMonth
dayYes; intDay
hourYes; intHour
minuteYes; intMinute
secondYes; intSecond
offsetYes; intTime offset related to time zone. Note: offset/100 = hours, offset %100 = minutes. Accepts values in the range <-1200,1400>.

Return value

The function returns a timestamp based on the date and time components passed in the arguments. The type of the value returned by the function is long. The resulting timestamp is a value in milliseconds. If the offset parameter is specified, the result will be returned shifted by the specified interval based on the passed parameters.

Example

ExampleResult
timestamp(2021, 6, 2, 17, 10, 3)1622653803000 (Wednesday, 2 June 2021 19:10:03 GMT+02:00 DST)
timestamp(2021, 6, 2, 18, 10, 3, offset=100) 1622653803000
timestamp(2021, 6, 2, 16, 10, 3, offset=-100)1622653803000

addYears

The function shifts the input timestamp by the number of years passed in the argument.

Syntax

addYears(long arg1, int arg2) : long

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; longTimestamp to be shifted.
arg2Yes; intThe number of years by which the timestamp from arg1 will be shifted

Return value

The function returns the timestamp from arg1 shifted by the number of years specified in arg2.
The type of the value returned by the function is long.
The resulting timestamp is a value in milliseconds.
The number of years can be positive - addition or negative - subtraction of years.

Example 1

For 1623069203000 (Monday, 7 June 2021 14:33:23 GMT+02:00 DST)

ExampleResult
addYears(1623069203000, 1)1686141203000 (Monday, 7 June 2020 14:33:23 GMT+02:00 DST)
addYears(1623069203000, -1)1559910803000 (Monday, 7 June 2022 14:33:23 GMT+02:00 DST)

addMonths

The function shifts the input timestamp by the number of months passed in the argument.

Syntax

addMonths(long arg1, int arg2) : long

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; longTimestamp to be shifted.
arg2Yes; intThe number of months by which the timestamp from arg1 will be shifted

Return value

The function returns the timestamp from arg1 offset by the number of months specified in arg2.
The type of the value returned by the function is long.
The resulting timestamp is a value in milliseconds.
The number of months can be positive - addition or negative - subtraction of months

Example 1

For 1623069203000 (Monday, 7 June 2021 14:33:23 GMT+02:00 DST)

ExampleResult
addMonths(1623069203000, 2)1628339603000 (Saturday, 7 August 2021 14:33:23 GMT+02:00 DST)
addMonths(1623069203000, -2)1559910803000 (Wednesday, 7 April 2019 14:33:23 GMT+02:00 DST)

addDays

The function shifts the input timestamp by the number of days passed in the argument.

Syntax

addDays(long arg1, int arg2) : long

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; longTimestamp to be shifted.
arg2Yes; intThe number of days by which the timestamp from arg1 will be shifted

Return value

The function returns the timestamp from arg1 shifted by the number of days specified in arg2. The type of the value returned by the function is long. The resulting timestamp is a value in milliseconds. The number of days can be positive - addition or negative - subtraction of days.

Example 1

For 1623069203000 (Monday, 7 June 2021 14:33:23 GMT+02:00 DST)

ExampleResult
addDays(1623069203000, 2) 1623242003000

addHours

The function shifts the input timestamp by the number of hours passed in the argument.

Syntax

addHours(long arg1, int arg2) : long

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; longTimestamp to be shifted.
arg2Yes; intThe number of hours by which the timestamp from arg1 will be shifted

Return value

The function returns the timestamp from arg1 shifted by the number of hours specified in arg2. The type of the value returned by the function is long. The resulting timestamp is a value in milliseconds. The number of hours can be positive - addition or negative - subtraction of hours.

Example 1

For 1623069203000 (Monday, 7 June 2021 14:33:23 GMT+02:00 DST)

ExampleResult
addHours(1623069203000, 2) 1623242003000

addMinutes

The function shifts the input timestamp by the number of minutes passed in the argument.

Syntax

addMinutes(long arg1, int arg2) : long

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; longTimestamp to be shifted.
arg2Yes; intThe number of minutes by which the timestamp from arg1 will be shifted

Return value

The function returns the timestamp from arg1 shifted by the number of minutes specified in arg2. The type of the value returned by the function is long. The resulting timestamp is a value in milliseconds. The number of minutes can be positive - addition or negative - subtraction of minutes.

Example 1

For 1623069203000 (Monday, 7 June 2021 14:33:23 GMT+02:00 DST)

ExampleResult
addMinutes(1623069203000, 2) 1623242003000

addSeconds

The function shifts the input timestamp by the number of seconds passed in the argument.

Syntax

addSeconds(long arg1, int arg2) : long

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; longTimestamp to be shifted.
arg2Yes; intThe number of seconds by which the timestamp from arg1 will be shifted

Return value

The function returns the timestamp from arg1 shifted by the number of seconds specified in arg2. The type of the value returned by the function is long. The resulting timestamp is a value in milliseconds. The number of seconds can be positive - adding or negative - subtracting seconds.

Example 1

For 1623069203000 (Monday, 7 June 2021 14:33:23 GMT+02:00 DST)

t = addSeconds(1623069203000, 2) 

Result:

t = 1623242003000  

Example 2

For 1623069203000 (Monday, 7 June 2021 14:33:23 GMT+02:00 DST)

ExampleResult
addSeconds(1623069203000, -2) 1623242003000

addMillis

The function shifts the input timestamp by the number of milliseconds passed in the argument.

Syntax

addMillis(long arg1, int arg2) : long

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; longTimestamp to be shifted.
arg2Yes; intThe number of milliseconds by which the timestamp from arg1 will be shifted

Return value

The function returns the timestamp from arg1 shifted by the number of milliseconds specified in arg2. The type of the value returned by the function is long. The resulting timestamp is a value in milliseconds. The number of milliseconds can be positive - addition or negative - subtraction of milliseconds.

Example 1

For 1623069203000 (Monday, 7 June 2021 14:33:23 GMT+02:00 DST)

ExampleResult
addMillis(1623069203000, 2)1623242003000
addMillis(1623069203000, -2) 1623242003000

getYear

The function retrieves the year from the timestamp passed in the argument.

Syntax

getYear(long arg1) : int

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; longTimestamp in epoch format in milliseconds

Return value

The function returns the year from the timestamp indicated in the argument.
The type of the value returned by the function is int.

Example 1

For 1623069203000 (Monday, 7 June 2021 14:33:23 GMT+02:00 DST)

ExampleResult
getYear(1623069203000)2021

getMonth

The function retrieves the month from the timestamp passed in the argument.

Syntax

getMonth(long arg1) : int

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; longTimestamp in epoch format in milliseconds

Return value

The function returns the month from the timestamp indicated in the argument.
The type of the value returned by the function is int being the month number of the year (1 - January, 2 - February, 3 - March, etc...).

Example 1

For 1623069203000 (Monday, 7 June 2021 14:33:23 GMT+02:00 DST)

ExampleResult
getMonth(1623069203000)6

getDay

The function retrieves the day of the month from the timestamp passed in the argument.

Syntax

getDay(long arg1) : int

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; longTimestamp in epoch format in milliseconds

Return value

The function returns the day of the month from the timestamp indicated in the argument.
The type of the value returned by the function is int.

Example 1

For 1623069203000 (Monday, 7 June 2021 14:33:23 GMT+02:00 DST)

ExampleResult
getDay(1623069203000)7

getDayOfWeek

The function retrieves the day of the week from the timestamp passed in the argument.

Syntax

getDayOfWeek(long timestamp) : int

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; longTimestamp in epoch format in milliseconds

Return value

The function returns the number of the day of the week from the timestamp specified in the argument.
The returned value is of int type with the following values: 1 - Monday, 2 - Tuesday, .... , 7 - Sunday.

Example 1

For 1623069203000 (Monday, 7 June 2021 14:33:23 GMT+02:00 DST)

ExampleResult
getDayOfWeek(1623069203000)1

getHour

The function retrieves the hours of the day from the timestamp passed in the argument.

Syntax

getHour(long timestamp) : int

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; longTimestamp in epoch format in milliseconds

Return value

The function returns the time of day from the timestamp indicated in the argument.
The type of the value returned by the function is int.

Example 1

For 1623069203000 (Monday, 7 June 2021 14:33:23 GMT+02:00 DST)

ExampleResult
getHour(1623069203000)14

getMinute

The function retrieves the minutes from the timestamp passed in the argument.

Syntax

getMinute(long timestamp) : int

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; longTimestamp in epoch format in milliseconds

Return value

The function returns minutes from the timestamp specified in the argument.
The type of the value returned by the function is int.

Example 1

For 1623069203000 (Monday, 7 June 2021 14:33:23 GMT+02:00 DST)

ExampleResult
getMinute(1623069203000)33

getSecond

The function retrieves seconds from the timestamp passed in the argument

Syntax

getSecond(long timestamp) : int

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; longTimestamp in epoch format in milliseconds

Return value

The function returns seconds from the timestamp specified in the argument.
The type of the value returned by the function is int.

Example 1

For 1623069203000 (Monday, 7 June 2021 14:33:23 GMT+02:00 DST)

ExampleResult
getSecond(1623069203000)23

tsToStr

The function converts the timestamp in long format to text of the specified format.

Syntax

tsToStr(long arg0 [, format=string] [, zone=string]) : string

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; longTimestamp in epoch format in milliseconds
formatNoFormat string to which the timestamp will be converted. The default value is: "YYYY-MM-dd HH:mm:ss.SSS".
zoneNoTime zone e.g.: +1, Poland, Europe/Warsaw, UTC. Description of possible formats: ZoneId (Java Platform SE 8 )

The function returns a String datetime representation in the format specified by kwarg format=string .
The representation will be offset in relation to the input timestamp by the kwarg zone=string.
If arg0 type different from long is passed, the function will return null.

Example 1

For 1623253047000 (Wednesday, 9 June 2021 17:37:27 GMT+02:00 DST)

ExampleResult
tsToStr(1623253047000)"2021-06-09 15:37:27.000"
tsToStr(1623253047000, format="YYYY/MM/dd-HH:mm:ss")"2021/06/09-15:37:27"
tsToStr(1623253047000, zone="Poland")"2021-06-09 17:37:27.000"
tsToStr("1623253047000", zone="Poland")null

tsParse

The function converts a textual representation of datetime to a timestamp in epoch format.

Syntax

tsParse(string arg0 [, format=string] [, zone=string]) : long

Parameters

Parameter NameRequired/not required; Field typeDescription
arg0Yes; stringDatetime text representation.
formatNo; stringFormat string used to interpret the passed timestamp. The default value is: "YYYY-MM-dd HH:mm:ss.SSS".
offsetNo; stringTime zone e.g.: +1, Poland, Europe/Warsaw, UTC. Description of possible formats: ZoneId (Java Platform SE 8 ).

Return value

The function returns a timestamp in epoch format of long type given in milliseconds.
The datetime text passed in arg0 will be interpreted as if it were in the time zone described in offset.
The returned result will always be in the GMT+00:00 zone.
If a value other than string is passed to arg0, the function will return null.

Example

ExampleResult
tsParse("2021-06-09 15:37:27.000")1623253047000
tsParse("2021/06/09-15:37:27", format="YYYY/MM/dd-HH:mm:ss")1623253047000
tsParse("2021-06-09 17:37:27.000", zone="Poland")1623253047000
1tsParse(1623253047000, zone="Poland")null

tsOffset

The function returns the timestamp offset specified in the argument.

Syntax

tsOffset(long arg0, int arg1) : long

Parameters

Parameter NameRequired/not required; Field typeDescription
arg0Yes; longTimestamp epoch given in milliseconds (to be shifted).
arg1Yes; intTime offset in relation to the time zone. Offset/100 = hours, offset %100 = minutes. Accepts values in the range of <-1200,1400>.

Return value

The function returns the timestamp offset value.
The function returns a value in epoch format of long type with millisecond precision.
If parameters of other types than expected are passed, the function will return null.

Example 1

For 1623253047000 (Wednesday, 9 June 2021 17:37:27 GMT+02:00 DST)

ExampleResult
tsOffset(1623253047000, 200)1623260247000 (Wednesday, 9 June 2021 19:37:27 GMT+02:00 DST)
tsOffset(1623260247000 , -200)1623253047000 (Wednesday, 9 June 2021 17:37:27 GMT+02:00 DST)
tsOffset(1623260247000 , -2200)null

sum

The function calculates the sum of num1 + num2 + ... + numN given in numeric value arguments. Non-numeric arguments are ignored.

Syntax

sum(number num1, number num2 [, number num3, ...object numN] [, type=string]) : number

Parameters

Parameter NameRequired/not required; Field typeDescription
num1Yes; number
num2Yes; number
numNNo; numberN-th argument of the function.
typeNo; numberForces the casting of each argument (numeric) to the specified type. Possible values: "int", "long", "float", "double". Determines the type of the returned value.

Return value

The function returns a value of Number type which is the result of adding all the arguments.
If the "type" parameter is not specified, the type of the value will be the same as the first numeric argument.
E.g. sum(null, "1", 1), the type returned will be int.
If all arguments are not numeric, 0 will be returned.

Example

ExampleResult
sum(2, 6)8 (typ int)
sum(2, 6, type="long")8 (typ long)
sum(null, 1, "4", 2)3
sum(1.1, 6)7.1 (float)
sum("1", "2")0 - because none of the arguments are of number type (int, long, float,...))

add

The function adds up the values specified in the arguments.

Syntax

add(number num1, number num2 [, type=string]) : number

Parameters

Parameter NameRequired/not required; Field typeDescription
num1Yes; number
num2Yes; number
typeNo; numberForces the casting of each argument (numeric) to the specified type. Possible values: "int", "long", "float", "double". Determines the type of the returned value.

Return value

The function returns a value of Number type which is the result of adding arguments.
The exact type is specified by the kwarg 'type', or when not specified, it is the type of the first argument.
Alternatively, if an argument other than Number is specified, a null value will be returned.
The function only accepts values of Number type. In case the kwarg 'type' does not specify the type to which arguments are to be cast, they will all be converted to the type specified in the first argument.

Example

ExampleResult
add(2, -6)-4
add(2, -6 type="long")-4 (typ long)
add(2, -0.1d)2 (int, the function will return a value of 1.9d rounded up to 2 (int) because that is the type of the first argument (2).)
add(2.0d, -0.1d)1.9 (double)

sub

The function subtracts the values specified in the arguments.

Syntax

sub(number num1, number num2 [, type=string]) : number

Parameters

Parameter NameRequired/not required; Field typeDescription
num1Yes; number
num2Yes; number
typeNo; numberForces the casting of each argument (numeric) to the specified type. Possible values: "int", "long", "float", "double". Determines the type of the returned value.

Return value

The function returns a value of Number type, which is the result of subtracting the arguments in the order they are given. The exact type is specified by the kwarg 'type', or when not specified, it is the type of the first argument.
Alternatively, if an argument other than Number is specified, a null value will be returned.

Example

ExampleResult
sub(2, 6)-4
sub(2, 6 type="long")-4 (typ long)

mul

The function performs the multiplication of two given arguments.

Syntax

mul(number num1, number num2 [, type=string]) : number

Parameters

Parameter NameRequired/not required; Field typeDescription
num1Yes; number
num2Yes; number
typeNo; numberForces the casting of each argument (numeric) to the specified type. Possible values: "int", "long", "float", "double". Determines the type of the returned value.

Return value

The function returns a value of Number type.
When the kwarg 'type' does not specify the type to which arguments are to be cast, they will be converted based on the type of the first argument as follows :
int → long
long → long
float → float
double → double
Alternatively, if an argument other than Number is specified, a null value will be returned.

Example

ExampleResult
mul(2, 6)12
mul(2, 2.1d)4
mul(2, 4.2d, type="double")8.4

div

The function performs the operation of dividing <2:n> values given in the arguments.

Syntax

div(number num1, number num2) : double

Parameters

Parameter NameRequired/not required; Field typeDescription
num1Yes; numberdividend
num2Yes; numberdivisor

Return value

The function returns the result of dividing the values specified in the arguments in the order they are given.
The type of the returned value is double.
If any of the arguments is 0, the function will return null.
Alternatively, if an argument other than Number is given, it will also return null.

Example

ExampleResult
sub(10.5, 2)5.25

floorDiv

The function divides without remainder the values given in the arguments.

Syntax

floorDiv(number num1, number num2) : long

Parameters

Parameter NameRequired/not required; Field typeDescription
num1Yes; numberdividend
num2Yes; numberdivisor

Return value

The function returns the result of dividing without remainder the values given in the arguments.
The type of the returned value is long.
If the divisor is 0, the function will return null.
If an argument other than Number is given, a null value will be returned.

Example

ExampleResult
floorDiv(17, 2.5)6
floorDiv(17, "a")null
floorDiv(17, 0)null

mod

The function calculates the remainder of one number divided by another.

Syntax

mod(object num1, object num2) : long

Parameters

Parameter NameRequired/not required; Field typeDescription
num1Yes; object
num2Yes; object

Return value

The function returns the remainder of dividing num1 argument by num2 argument.

The type of the returned value is long.

If a value other than Number is specified in the arguments, it will return null.

Example

ExampleResult
mod(10, 6)4

abs

The function calculates the absolute value for the given argument.

Syntax

abs(number num) : number

Parameters

Parameter NameRequired/not required; Field typeDescription
num1Yes; number

Return value

The function returns an absolute value for the specified argument with the same type.
If an argument with a type different than Number is specified, it will return null.

Example

ExampleResult
abs(-4)4

pow

The function calculates powers for specified arguments.

Syntax

pow(number num1, number num2) : double

Parameters

Parameter NameRequired/not required; Field typeDescription
num1Yes; numberPower base.
num1Yes; numberThe power to which the base will be raised.

Return value

The function returns the value of the first argument raised to a power whose value is specified in the second argument.
The returned type is double.
Alternatively, if a value other than Number is specified in the arguments, a null value is returned.

Example

ExampleResult
pow(2, 6)64

sqrt

The function calculates the square root of the number specified in the argument.

Syntax

sqrt(number num) : double

Parameters

Parameter NameRequired/not required; Field typeDescription
num1Yes; number

Return value

The function returns the value of the square root of the number specified in the argument. The type of the returned value is double. If a value other than Number is given in the argument, it will return null. If you specify a number value less than or equal to 0 - it will also return null.

Example

ExampleResult
sqrt(2)4
sqrt(-2)null

sqr

The function squares the number specified in the argument.

Syntax

sqr(number num) : double

Parameters

Parameter NameRequired/not required; Field typeDescription
num1Yes; number

Return value

The function returns the square of the number specified in the argument.
The returned type is double.
If you pass an argument with a type other than Number, the function will return null.

Example

ExampleResult
sqr(2)4.0 (double)
sqrt(-2)null

floor

The function is used to round down the number passed in the argument to the integer part.

Syntax

floor(number num) : long

Parameters

Parameter NameRequired/not required; Field typeDescription
num1Yes; number

Return value

The function returns the rounded down numeric value of the parameter specified in the argument.
It rounds down to an integer.
If the specified parameter values are other than Number, the function will return null.

Example

ExampleResult
floor(1.2)1
floor(3.54)3

ceil

The function rounds up the number passed in the argument.

Syntax

ceil(number num) : long

Parameters

Parameter NameRequired/not required; Field typeDescription
num1Yes; number

Return value

The function returns the rounded up numeric value of the parameter specified in the argument.
It rounds up to an integer.
If the specified parameter value is other than Number, the function will return null.

Example

ExampleResult
ceil(1.2)2
ceil(3.54)4

sign

The function checks whether the value in the argument is a positive, zero or negative value.

Syntax

sign(number num) : int

Parameters

Parameter NameRequired/not required; Field typeDescription
num1Yes; number

Return value

The function returns the following values of int type:
-1 - the passed value is negative
0 - the passed value is zero
1 - the passed value is positive
null - the passed value is not of numeric type

Example

ExampleResult
sign(3)1
sign(-3)-1
sign("3")null

rand

The function generates a random number in the range (0,1).

Syntax

rand() : double

Parameters

Parameter NameRequired/not required; Field typeDescription

Return value

The function returns a pseudo-random value of double type in the range from 0 to 1.

Example

ExampleResult
rand()0.5072899523834322

randInt

The function returns an integer random number in the specified range.

Syntax

randInt(int arg0, int arg1) : int

Parameters

Parameter NameRequired/not required; Field typeDescription
arg0Yes; intThe beginning of the range
arg1Yes; intThe end of the range

Return value

The function returns a pseudo-random number of int type in the range <arg0,arg1>.

Example

ExampleResult
randInt(1,5)3

bitAnd

The function performs the "and" operation on numeric arguments. Non-numeric arguments are ignored.

Syntax

bitAnd(number num1, number num2 [, number num3, ...object numN]) : number

Parameters

Parameter NameRequired/not required; Field typeDescription
num1..NYes; numberNumber

Return value

The function returns a value of number type which is the result of the "and" operation for all numeric arguments.
If all arguments are not numeric, it will return null.

Example

ExampleResult
bitAnd(1, 5)3
bitAnd(5, "0")5
bitAnd("5", "5")null

bitOr

The function performs "or" operations on numeric arguments. Non-numeric arguments are ignored.

Syntax

bitOr(number num1, number num2 [, number num3, ...object numN]) : number

Parameters

Parameter NameRequired/not required; Field typeDescription
num1,...,numNYes; numberNumber

Return value

The function returns a value of number type , which is the result of the "or" operation for all numeric arguments.
If all arguments are not numeric, it will return null.

Example

ExampleResult
bitOr(1, 2)3
bitOr(1, "2"1
bitOr("1", "2")null

if

Conditional function if.

Syntax

if (condition : result1 , result2) 

Parameters

Parameter NameRequired/not required; Field typeDescription
conditionYes; logical valueLogical condition for the selection of result 1.
result1YesThis result is returned if the result of the logical condition is true.
result2YesThis result is returned if the result of the logical condition is false.

Return value

If the result of the "condition" condition is "true" (true), the function will return the value of the "result1" expression, otherwise (false) - "result2".

Example 1

if (fName="Jack": "This is Jack", "This is not Jack") 

Result:

If the value of the fName field is equal to "Jack":

"This is Jack"

If the value of the fName field is not equal to "Jack":

"This is not Jack"

switch

The function returns the result depending on the condition.

Syntax

switch (variable, variable_value_1:result1, variable_value_2:result2, default_result)

Parameters

Parameter NameRequired/not required; Field typeDescription
variableYes;
variable_value_1:result1Yes;
variable_value_2:result2Yes;
default_resultYes;

Return value

If the value of the variable argument is equal to: -- variable_value_1 then the function returns "result1" -- variable_value_2 then the function returns "result2" -- different from 1 and 2, then the function returns "default_result"

Example 1

switch(ctry, "DE": "Germany", "IT": "Italy", "Other")

Result:

If the value of the ctry field is equal to "DE":

"Germany"

If the value of the ctry field is equal to "IT":

"Italy"

If the value of the ctry field is not equal to "DE" and "IT"

"Other"

concat

The function concatenates at least two values into one string.

Syntax

concat(object arg1 , object arg2 [, ... object argN] [, delimiter=string distinct=boolean limit=integer]) : string

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; object
arg2Yes; object
argNNo; objectThe function accepts N arguments.
delimiterNo; stringSeparator of concatenated strings. Default value: ""
distinctNo; booleanMerge only unique values. default value: false
limitNo; intLimit of combined values. Default value: Integer.MAX_VALUE

Return value

The function returns a String object which is a concatenation of the passed arguments in the order they are given.

Example 1

ExampleResult
concat("abc", "-", "def")"abc-def"
concat("1", "2", delimiter=";")"1;2"
concat("1", 1, "2", distinct=true)"12"
concat("1", "2", "3", limit=2)"12"
concat("1", 1, "2", "3", distinct=true limit=2)"12"

substring

The function cuts sequences of characters from the String object and creates a new object from them.

Syntax

substring(object arg, int num1 [, endId=num2]) : string

Parameters

Parameter NameRequired/not required; Field typeDescription
argYes; objectInput object, on the basis of which a new object will be created.
num1Yes; intIndex of the character sequence from which the new object will start. The character located at the indicated index will be contained in the result.
num2No; intThe final index of the character sequence. The character located at the indicated index will not be included in the result.

Return value

The function returns a new String that is a fragment of the input String object (or converted to String) based on the arguments indicating the index range.
The function allows the use of negative indexes that point to an index counting from the end of the String.
If an initial index higher than the final index is indicated, the function will return null.
If indexes outside the range are indicated, the function will return null.

ExampleResult
substring("Cat is wild.", 4, endId=6)"is"
substring("Cat is wild.", 5, endId=4)null
substring("Cat is wild.", 12, 13)null

left

The function cuts the beginning of the character sequence from the passed object.

Syntax

left(object arg, int num) : string

Parameters

Parameter NameRequired/not required; Field typeDescription
argYes; objectInput String object.
numYes; intString object cut index from the left.

Return value

The function extracts and returns a num number of characters from a arg string (starting from left).

The function allows the use of negative num value, which point to the index counting from the end of the string.

Example 1

ExampleResult
left("Cat is wild.", 6)"Cat is "
left("Cat is wild.", -2)"Cat is wil"
left(512, 1)"51"

The function extracts and returns a num number of characters from a arg string (starting from right).

Syntax

right(object arg, int num) : string

Parameters

Parameter NameRequired/not required; Field typeDescription
argYes; objectInput String object.
numYes; intIndex of String object cut off from the right side

Return value

The function returns a new String object based on the object passed in the arg argument, cutting it off at index num inclusive. The right side of the String object is returned.
The function allows the use of negative indexes, which indicate the index counting from the end of the string, where -1 indicates the last character.

Example 1

ExampleResult
right("Cat is wild.", 5)"wild."
right(512, 1)"12"

lower

The function replaces uppercase letters in a string with lowercase letters.

Syntax

lower(object arg) : string

Parameters

Parameter NameRequired/not required; Field typeDescription
argYes; objectInput String object

Return value

The function returns the input string converted to lowercase.
If a number type is passed, the corresponding text representation will be returned.

ExampleResult
lower("CAT is WILD.")"cat is wild."

upper

The function replaces lowercase letters in a string with uppercase letters.

Syntax

upper(object arg) : string

Parameters

Parameter NameRequired/not required; Field typeDescription
argYes; objectInput String object

Return value

The function returns the input string converted to uppercase.
If a number type is passed, the corresponding text representation will be returned.

Example 1

ExampleResult
lower("CAT is WILD.")"CAT IS WILD."

replace

The function replaces a string of characters with another string of characters.

Syntax

replace(object arg, string str1, string str2 [, strat=string]) : string

Parameters

Parameter NameRequired/not required; Field typeDescription
argYes; objectInput String object.
str1Yes; stringThe string to be found and replaced.
str2Yes; stringThe string that will be put in place of str1
stratNo; stringReplacement strategy. The available options are: "first", "last", "all". The default value is "first".

Return value

The function returns a new String object that is a modification of the input object.
If a number type is passed in arg, the corresponding text representation will be processed.
If the str1 string is not found, the unprocessed text representation passed in arg will be returned.

Example 1

ExampleResult
replace("Cat is wild.", "wild", "nice") "Cat is nice."
replace("Cat is wild and wild", "wild", "nice", strat="last")"Cat is wild and nice."

trim

The function removes whitespace characters from the beginning and end of a string.

Syntax

trim(string arg) : string

Parameters

Parameter NameRequired/not required; Field typeDescription
argYes; stringInput String object.

Return value

The function returns a new String object, created from the input object without whitespace characters at the beginning and end of the string.
If a value other than String is passed, the function will return null.

Example 1

ExampleResult
trim("Cat is wild.")"Cat is wild."
trim(" Cat is wild. ")"Cat is wild."

startsWith

The function verifies if the object begins with the string passed in the next argument.

Syntax

startsWith(object arg1, object arg2) : boolean

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; objectCompared object.
arg2Yes; objectThe string that will be searched for at the beginning of arg1.

Return value

The function returns true if the string from arg2 is the same as the string from the beginning of arg1.
If a number type is passed in the arguments, the corresponding text representation will be processed.

Example 1

ExampleResult
startsWith("Cat is wild.", "Cat")true
startsWith("Cat is wild.", "Dog")false
startsWith(512, "5")true

endsWith

The function verifies if the object ends with the string passed in the next argument.

Syntax

endsWith(object arg1, object arg2) : boolean

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; objectCompared object.
arg2Yes; objectThe string that will be searched for at the beginning of arg1.

Return value

The function returns true if the string from arg2 is the same as the string from the end of arg1.
If a number type is passed in the arguments, the corresponding text representation will be processed.

Example 1

ExampleResult
endsWith("Cat is wild.", "wild.")true
endsWith("Cat is wild.", "is")false
endsWith(512, "2") true

contains

The function checks if a specified string exists in the other string.

Syntax

contains(object arg1, object arg2 [, ignorecase=boolean]) : boolean

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; objectThe string on which the check will be performed.
arg2Yes; objectThe string that will be searched for in arg1.
ignorecaseNo; booleanFlag - whether the check should be case sensitive

Return value

The function returns true if the string from arg2 occurs inside the string of arg1 at least once.
If a number type is passed in the arguments, the corresponding text representation will be processed.

Example 1

ExampleResult
contains("Cat is wild.", "Cat")true
contains("Cat is wild.", "Dog")false
contains("Cat is wild.", "cat", ignorecase=true)true
contains("512421", 241)true

indexOf

The function searches for a substring in a string, and returns the position.

Syntax

indexOf(object arg1, object arg2 [, startIndex=int] [, ignoreCase=boolean]) : int

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; objectThe object in which the character will be searched for
arg2Yes; objectThe character that will be searched for
startindexNo; intThe character string index from arg1 from which the search will be started. The default value is 0. The specified index also undergoes the search.
ignorecaseNo; booleanFlag stating if the comparison is supposed to take into consideration the character size.

Return value

The function returns a number value of int type corresponding to the first character string index from arg1 which is equal to the character passed in arg2.

If the character is not found, the function will return null.

In case of passing in number type arguments, a corresponding text representation will be processed.

Example 1

ExampleResult
indexof("Cat is wild.", "C")0
indexof("Cat is wild.", "t")2
indexof("Cat is wild.", "i", startindex=6)8
indexof("Cat is wild.", "c", ignorecase=true)0
indexof("Cat is wild.", "b")null

Example 6

src stream="testdata" | fIndexOf1 = indexOf(fName, "A", startIndex=1 ignoreCase=true),fIndexOf2 = indexOf(fName, "A", ignoreCase=false) | limit 1

Result:

{
"lName": "White",
"fName": "Jake",
"fIndexOf1": 1,
"fIndexOf2": null
}

strlen

The function returns the length of the text representation of the passed character string in the argument.

Syntax

strlen(object arg) : int

Parameters

Parameter NameRequired/not required; Field typeDescription
argYes; object

Return value

The function returns the length of the text representation of the passed character string in the argument

In case of passing in number type arguments, a corresponding text representation will be processed.

Example 1

ExampleResult
len("Cat is wild.")12
len(5)1

regex

The function allows for usage of regular expressions. The function returns a logical value (boolean) of searching for a pattern (second parameter) in the text (first parameter). The function can also convert various non-string object types. When it does, it uses the toString() conversion.

Syntax

regex(object arg1, object arg2) : boolean

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; objectsearched text / field
arg2Yes; objectpattern, regular expression

Return value

The function returns a true/false value that is the result of a search for a pattern in the text.

Example 1

ExampleResult
regex("Postal code 02-120", "\\d{2}-\\d{3}") true
`regex("automatic transmission", "auto(matbus)" )`

like

The function mimics the database (SQL) function of the same name. It returns a logic value (boolean) which is the result of searching for a pattern in the searched text.

Syntax

like(textOrField, pattern) : boolean

Parameters

Parameter NameRequired/not required; Field typeDescription
textOrFieldYes; objectsearched text / field
patternYes; objectpattern, containing the characters: "*" (any string of characters) and "." (one character).

Return value

The function returns the value true/false, which is the result of searching for a pattern in the text. The pattern can contain one or more characters: "*" (any string of characters) and "." (one character),

Example 1

ExampleResult
like("*co*", "Sycope")true
like(".yco*", "Sycope")true
like("s.u*", "passus")false
like("s.u*", "ssus")true

isIp

The function verifies if the passed object is a valid representation of an IPv4 or IPv6 address version.

Syntax

isIp(object arg0) : boolean

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; objectThe passed parameter can be of String type, IpAddress or int[].

Return value

The function returns true if the object passed in the argument is a valid representation of an IPv4 or IPv6 address.

Example 1

ExampleResult
isIp("172.16.40.1")true
isIp("172.256.1.1")false
isIp("ffe0::1")true
isIp(int[]{1,2,3,4})false

toIp

The function creates an IP object based on the passed text representation.

Syntax

toIp(string agr1 [, type=string]) : object

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; stringText that will be converted to an object representation of the IP address.
typeNo; stringThe type of the returned object. The available options are: Array and IpAddress. By default, it returns int[] (Array).

Return value

The function returns an object representation of the IP address in the format specified by the kwarg type. śThe function returns an int[] object if the argument type is not specified.

Example 1

ExampleResult
toIp("172.16.40.1")int[]{2886739969}
toIp("172.16.40.1", type=IpAddress)value of type IpAddress
toIp("::1", type=Array)int[]{0,0,0,1}

toMac

The function creates a MAC Address object based on the passed text representation.

Syntax

toMac(string arg1 [,type=string]) : object

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; stringText that will be converted to an object representation of the MAC address. Delimiters are allowed in the form of: ":" and "-"
typeNo; stringThe type of the returned object. The available options are: Array and MACAddress. By default, it returns a byte[] (Array).

Return value

The function returns an object representation of the MAC address in the format specified by the kwarg type.
The function returns a byte[] object if the argument type is not specified.
If an incorrect representation of the MAC address is specified, the function will return null.

Example 1

ExampleResult
toMac("01:02:03:FA:BD:CE")(byte[]) [1, 2, 3, -6, -67, -50]
toMac("01-02-03-FA-BD-CE", type=MACAddress)value of type MACAddress
toMac("01:02:03:FA:BD:CE", type=Array)null
toMac("01:02:03:FA:BD:FG")null

isMac

The function verifies if the passed object is a valid representation of the MAC address.

Syntax

isMac(object arg1) : boolean

Parameters

Parameter NameRequired/not required; Field typeDescription
arg1Yes; booleanThe passed parameter can be of String type, MACAddress or byte[].

Return value

The function returns true if the object passed in the argument is a valid representation of the MAC address.

Example 1

ExampleResult
isMac("AA:BB:CC:DD:EE:FF")true
isMac(byte[]{1, 2, 3, -6, -67, -50})true
isMac("FG:00:00:00:00:00")false

ipToStr

The function returns a text representation of the IP address.

Syntax

ipToStr(object arg1 [,short=boolean]) : string

Parameters

NameRequired, TypeDescription
arg1yes, objectIP address representation in int[], IpAddress or String format.
arg1no, booleanFlag that sets the shortened or full address record in IPv6. The default value is false. For IPv4, the value is ignored.

Return value

The function returns a text representation of the IPv4 or IPv6 address passed in the argument.
If an invalid IP address is passed, the function returns null.

Example 1

ExampleResult
ipToStr(int[]{2886739969})172.16.40.1
ipToStr(int[]{1})"0.0.0.1"
ipToStr(int[]{0,0,0,1}, short=true) "::1"
ipToStr(int[]{1,2})null

macToStr

The function returns a text representation of the MAC address.

Syntax

macToStr(object arg1) : string

Parameters

NameRequired, TypeDescription
arg1yes, objectMAC address in String, byte[] or MACAddress format

Return value

The function returns a text representation of the MAC address. Returns null if arg1 has an invalid value/format.

Example

ExampleResult
macToStr("01:02:03:FA:BD:CE")"01:02:03:FA:BD:CE"
macToStr(byte[]{1, 2, 3, -6, -67, -50})"01:02:03:FA:BD:CE"

toSubnet

The function creates an IpSubnet object from a text representation passed in the arg1 parameter.

Syntax

toSubnet(string arg1) : IpSubnet

Parameters

NameRequired, TypeDescription
arg1yes, stringSubnet mask in "ip/cidr" format.

Return value

Returns an IpSubnet object created from a text representation. If arg1 value is invalid the function will return null.

Example

ExampleResult
toSubnet("172.16.40.0/24")IpSubnet value

isSubnet

The function verifies if the arg1 IPv4 or IPv6 address is a valid representation of the net mask for version 4 or 6.

Syntax

isSubnet(object arg1) : boolean

Parameters

NameRequired, TypeDescription
arg1yes, objectThe passed argument should be a correct representation of the subnet mask of String type or IpSubnet.

Return value

Returns true if arg1 IPv4 or IPv6 address is a valid representation of the net mask for version 4 or 6.

Example

ExampleResult
isSubnet("172.16.40.0/24")true
isSubnet("ffff::/129")false

inSubnet

The function checks if the ip address belongs to the subnet subnet mask.

Syntax

inSubnet(object ip, object subnet) : boolean

Parameters

NameRequired, TypeDescription
ipyes, objectThe IPv4 ot IPv6 address in String, int[] or IpAddress format.
subnetyes, objectThe net mask in String or IpSubnet format.

Return value

Returns true if the ip address belongs to the subnet subnet mask and both arguments have valid value. Returns false if the ip address does not belong to the subnet subnet mask. Returns null if one of the arguments has invalid value.

Example

ExampleResult
inSubnet("172.16.40.1", "172.16.40.0/24") true
inSubnet("172.16.40.1", "172.16.30.0/24")false
inSubnet("172.16.40.1", "aaa")null

subnetToStr

The function returns a text representation of the net mask.

Syntax

subnetToStr(object arg1) : string

Parameters

NameRequired, TypeDescription
arg1yes, objectThe IpSubnet object or text of a net mask.

Return value

Returns a text representation of the net mask. In case of IpSubnet type in the arg parameter, the function returns a string type. In case of String type in the arg parameter, the function validates the value and returns a string type or returns null if the validation failed.

Example

ExampleResult
subnetToStr(IpSubnet("172.16.40.0/24"))"172.16.40.0/24"
subnetToStr("172.16.40.0/24")"172.16.40.0/24"
subnetToStr("172.16.40.256/24")null

lookup

The function returns an outputKey field value of the first found object in the lookupName lookup which meets the criteria defined in the input.

Lookup elements are used for getting a data from external sources (file, database, DNS server etc.)

The data included in the lookups can be imagined as a list of records containing pairs of keys and values or as a table where the columns correspond to keys and the rows correspond to records.

In order to gain access to a specific value, the key (column) needs to be provided together with the criteria of record (row) selection in the form of pair: key, expected value

The lookup elements can also answer the question if a record with given criteria has any occurrence, and they do so without loading any values from them.

Please note that the function is an intermediate for the lookup and the returned value usually reflects the value returned by the same lookup ([L]). The function decides on the returned value [F] only in certain situations.

Syntax

lookup(string lookupName, string output, Map<string, object> input[,[default=string][defaultToKey=boolean][unwind=field]]) : object

Parameters

NameRequired, TypeDescription
lookupNameyes, stringLookup name.
outputKeyyes, stringField name of an object (lookup row).
inputyes, mapSearch criteria in Key-value collection format.
defaultno, stringA value which is returned when no objects were found by the criteria. The parameter has a higher priority on DefaultKey parameter.
defaultToKeyno, booleanIf true then the function returns a key value in case if no objects were found. If false, the parameter is ignored.
unwindNo; stringIt is used to handle input fields of Set/List type. The field name must be indicated in the declaration. The result of the function will be a list, in which each element will be the result of a lookup function set on consecutive list elements of the indicated field.

Return value

The outputKey field value of the first object in the lookupName lookup found by criteria defined in input.

Example

This example presents a lookup named "geoIp1" which contains two columns, "ip" and "country".

ExampleResult
lookup("geoIp1", "country", {"ip": "212.77.98.8"})"PL"
lookup("geoIp1", "country", {"ip": "127.0.0.1"})null
lookup("geoIp1", "country", {"ip": ipField})country code or null
lookup("snmpInterface", "if.speed", {"exporterIp": "172.16.40.1", "exporterInt": 9})2000000000
lookup("noSuchLookup", "country", {"ip": ipField})null
lookup("geoIp1", "noSuchColumn", {"ip": ipField})null
lookup("geoIp1", "country", {"noSuchColumn": ipField})null
lookup("groups", "name", {"ip": "172.16.0.1"}, default="no result ")"private-group"
lookup("groups", "name", {"ip": "1.2.3.4"}, default="no result ")"no result"
lookup("groups", "name", {"ip": "1.2.3.4"}, default="no result " defaultToKey=true)"no result"
lookup("groups", "name", {"ip": "1.2.3.4"}, defaultToKey=true)"1.2.3.4"
lookup("groups", "name", {"ip": ["1.2.3.4", "172.16.0.1"]}, defaultToKey=true unwind="ip")["1.2.3.4", "private-group"]
lookup("groups", "name", {"ip": ["1.2.3.4", "172.16.0.1"]}, unwind="ip")[null, "private-group"]

General Example

For this example, a lookup called "books" with the following data will be used:

authortitleyearISBN
Brian GoetzJava Concurrency in Practice20060321349606
Joshua BlochEffective Java (3rd edition)20180134685991
Scott OaksJava Performance: The Definitive Guide20141449358457
Scott OaksJava Performance: In-depth Advice for Tuning and Programming Java 8, 11, and Beyond20201492056119
ExampleResult
lookup("books", "ISBN", {"author": "Gavin King"})null (no match)
lookup("books", "ISBN", {"author": "Scott Oaks"})1449358457 (first match)
lookup("books", "ISBN", {"author": "Scott Oaks", "year": 2020})1492056119
lookupAll("books", "ISBN", {"author": "Gavin King"})[] (no match)
lookupAll("books", "ISBN", {"author": "Brian Goetz"})[0134685991]
lookupAll("books", "ISBN", {"author": "Scott Oaks"})[1449358457, 1492056119]
lookupKeyExists("books", {"author": "Gavin King"})false
lookupKeyExists("books", {"author": "Scott Oaks"})true

lookupAll

The function returns an outputKey field value of all objects in the lookupName lookup which meets the criteria defined in the input.

Syntax

lookupAll(string lookupName, string outputKey, Map<string, object> input[, default=string, defaultToKey=boolean]) : object

Parameters

NameRequired, TypeDescription
lookupNameyes, stringLookup name.
outputKeyyes, stringField name of an object (lookup row).
inputyes, mapSearch criteria in Key-value collection format.
defaultno, stringA value which is returned when no objects were found by the criteria. The parameter has higher priority on the DefaultKey parameter.
defaultToKeyno, booleanIf true then function will return a key value in case if no objects were found. If false, the parameter is ignored.

Return value

The outputKey field value of all objects in the lookupName lookup which meet the criteria defined in the input. The function returns a maximum of 1000 objects. If no objects were found, the function will return an empty value. If a lookup is turned off or does not exist, the function will return null.

Example

This example presents a lookup named "geoIp1" which contains two columns, "ip" and "country".

ExampleResult
lookupAll("geoIp1", "country", {"ip": "212.77.98.8"})["PL"]
lookupAll("geoIp1", "country", {"ip": "127.0.0.1"})[]
lookupAll("groups", "name", {"ip": "127.0.0.1"})["loopback", "private", "forbidden"]
lookupAll("groups", "name", {"ip": "1.1.1.1"}, default="no results")["no results"]
lookupAll("groups", "name", {"ip": "1.1.1.1"}, default="no results" defaultToKey=true)["no results"]
lookupAll("groups", "name", {"ip": "1.1.1.1"}, defaultToKey=true)["1.1.1.1"]

lookupKeyExists

The function checks if an occurrence of an object in the lookupName lookup meets the input criteria.

syntax

lookupKeyExists(string lookupName, Map<string, object> input) : boolean

Parameters

NameRequired, TypeDescription
lookupNameyes, stringA lookup name.
inputyes, mapSearch criteria in Key-value collection format.

Return value

The function returns true if an object with given criteria has been found in the lookup, false if otherwise.

Example

This example presents a lookup named "geoIp1" which contains two columns, "ip" and "country".

ExampleResult
lookupKeyExists("geoIp1", {"ip": "212.77.98.8"})true
lookupKeyExists("geoIp1", {"ip": "127.0.0.1"})false
lookupKeyExists("geoIp1", {"ip": ipField})true
lookupKeyExists("noSuchLookup", {"ip": ipField})null
lookupKeyExists("geoIp1", {"noSuchColumn": ipField})false

in

The function checks if an arg0 value is in the arg1 collection.

Syntax

in(object arg1, string arg2) : boolean

Parameters

NameRequired, TypeDescription
arg1yes, objectA value
arg2yes, stringCollection name.

Return value

Returns true if the arg1 collection contains an arg value, false if otherwise.

Example

ExampleResult
in(1, [1,2,3])true
in("mary", ["has", "cat", "mary"])true
in("a", ["b", "c"])false

all

The function checks if an arg0 collection is a subset of the arg1 collection.

Syntax

all(string arg0, string arg1) : boolean

Parameters

NameRequired, TypeDescription
arg0yes, stringCollection name.
arg1yes, stringCollection name.

Return value

Returns true if the arg0 collection is a subset of the arg1 collection, false if otherwise.

Example

ExampleResult
all(["a", "b"], ["a", "b", "c"])true
all(["a", "d"], ["a", "b", "c"])false

any

The function checks if there are any common values in both collections.

Syntax

any(string arg0, string arg1) : boolean

Parameters

NameRequired, TypeDescription
arg0yes, stringCollection name.
arg1yes, stringCollection name.

Return value

Returns true if the collections passed in the function parameters do have common values, false if otherwise.

Example

ExampleResult
any(["value1"], ["value2", "value3"])false
any(["value2"], ["value2", "value3"])true

none

The function checks if there are no common values in both collections.

Syntax

none(string arg0, string arg1) : boolean

Parameters

NameRequired, TypeDescription
arg0yes, stringCollection name.
arg1yes, stringCollection name.

Return value

Returns true if the collections passed in the function parameters do not have any common values, false if otherwise.

Example

ExampleResult
none(["value1"], ["value2", "value3"])true
none(["value2"], ["value2", "value3"])false

len

The function returns the number of elements in a collection (array, list, set).

Syntax

len(collection arg0) : number

Parameters

NameRequired, TypeDescription
arg0Yes; collectionCollection or table

Return value

Returns the number of elements in the collection or null when the argument is not a collection.

Example

ExampleResult
len([1, "two", 3.0]) 3
len("test")null

valueAt

The function returns a value at the index index in the arg1 collection.

Syntax

valueAt(string arg1, int index): object

Parameters

NameRequired, TypeDescription
arg1yes, stringCollection name.
arg1yes, intIndex

Return value

The function returns a value at the index position started from 0 in the arg1 collection or null if an index is out of range or arg1 is not a collection.

ExampleResult
valueAt([1, "two", 3.0], 2)3
valueAt([1, "two", 3.0], 3) )null
valueAt("Feigenbauma const", 0) null

hasKey

The function checks if the map contains a key.

Syntax

hasKey(string key, map map): boolean

Parameters

NameRequired, TypeDescription
keyYes; stringA key to test.
mapYes; mapCollection (map) name.

Return value

Returns true if a key exists in the keys set of the map, false if otherwise.

Example 1

ExampleResult
hasKey("a", {"a": 1})true
hasKey("a", {"b": 1})false
hasKey("a", {"a": null})true
hasKey(null, {"a": 1})false

hasValue

The function checks if a value value exists in the map.

Syntax

hasValue(string value, map): boolean

Parameters

NameRequired, TypeDescription
valueyes, stringA value to test.
mapyes, mapCollection (map) name.

Return value

Returns true if a value exists in the values set of the map, false if otherwise.

Example 1

ExampleResult
hasValue(1, {"a": 1})true
hasValue(2, {"a": 1})false
hasValue(null, {"a": 1})false
hasValue(null, {"a": null})true

get

The function returns a value from the collection at a given index.

Syntax

get(int index, collection) : object

Parameters

NameRequired, TypeDescription
indexyes, intindex
collectionyesCollector name (identifier).

Return value

The function returns a value from the collection at a given index or null if the index is out of range (index < 0 or index > collection length).

Example 1

ExampleResult
get(0, [10,20,30])10
get(1, [10,20,30])20
get(-1, [10,20,30])null
get(4, [10,20,30])null
get(0, [null,20,30])null

getValue

The function returns a value assigned to the key in the map map.

Syntax

getValue(string key, map): boolean

Parameters

NameRequired, TypeDescription
keyyes, stringKey name
collectionyes, mapA map contains key-value pairs.

Return value

The function returns a value assigned to the key in the map or null if the key does not exists in the map.

Example 1

ExampleResult
getValue("a", {"a": 1})1
getValue("b", {"a": 1})null
getValue("a", {"a": null})null
getValue(null, {"a": 1})null

anyMatchingNumber

The function checks if there is an occurrence of a value in the collectorId collection.

Syntax

anyMatchingNumber(string collectorId, long mask, long arg2): boolean 

Parameters

NameRequired, TypeDescription
collectorIdyes, stringCollector name (identifier).
maskyes, long
arg2yes, long

Return value

Returns true if there is at least one value in the collection that meets the requirement: (value & mask) == arg2 where (value & mask) is a result of AND bits operation.


noMatchingNumber

The function checks if there is an occurrence of a value value in the collectorId collection.

Syntax

noMatchingNumber(string collectorId, long mask, long arg2): boolean 

Parameters

NameRequired, TypeDescription
collectorIdyes, collectionCollector name (identifier).
maskyes, longMask to calculate value to search for.
arg2yes, longA value to search.

Return value

Returns true if there is no value in the collection that meets the requirement: (value & mask) == arg2 where (value & mask) is a result of AND bits operation. The result is opposite to the anyMatchingNumber function (noMatchingNumber := NOT anyMatchingNumber).


valInColl

The function checks if a value exists in the fieldName field in the collectorId collector.

Syntax

valInColl(object value, string collectorId, string fieldName [, maxCacheSize=<integer>]) : boolean

Parameters

NameRequired, TypeDescription
valueyes, objectA value to be checked.
collectorIdyes, stringCollector name (identifier).
fieldNameyes, stringA field name in collector to be search for.
maxCacheSizeno, intThe max. number of results (objects) kept in the cache. The 0 value means that the cache is not used. Default: 0.

Return value

Example 1

For the values in "coll1" collector:

[{"field":1},{"field":2},{"field":3.0},{"field":null}]

ExampleResult
valInColl(1, "coll1", "field")true
valInColl(99, "coll1", "field")false

collExists

The function checks if a collectorId collector exists.

Syntax

collExists(string collectorId) : boolean

Parameters

NameRequired, TypeDescription
collectorIdyes, stringCollector name (identifier).

Return value

Returns true if the collector exists, false if otherwise.

Example

ExampleResult
collExists("hosts")false

valColl

The function returns a value from the input field of an object found by the input criteria in the collector Name collector. The function can be used to join values from two or more collectors.

Syntax

valColl(String collectorName, String output, Map<String, Object> input [,default=String])

Parameters

NameRequired, TypeDescription
collectorNameyes, stringCollector name
outputyes, stringField name.
inputyes, mapA criteria in key-pair format.
defaultno, stringA default value returned if no object was found by the given criteria.

Return value

The function returns the value from the first object from the collector found by the criteria.

Example 1

In this example it is assumed that a "qwerty" collector is present with the following data:

[
{ "x": 1, "y": 10, "z": "a" },
{ "x": 1, "y": 20, "z": "b" }
]
ExampleResult
valColl("qwerty", "y", {"x": 1})10
valColl("qwerty", "y", {"x": 1, "z": "b"})20
valColl("qwerty", "y", {"x": -1234567})null
valColl("qwerty", "y", {"x": -1234567}, default=999)999
valColl("qwerty", "noSuchColumn", {"x": 1})null
valColl("qwerty", "noSuchColumn", {"x": 1}, default=999)999

join

The function joins values of a field into one string.

Syntax

join(field [, delimiter=string distinct=boolean limit=integer]) : string

Parameters

NameRequired, TypeDescription
fieldyes, stringThe field name whose values are to be joined.
delimiterno, stringThe values delimiter. Default: ","
distinctno, booleantrue - join only unique values. Default: false
limitno, intThe max. number of joined values. Default: Integer.MAX_VALUE

Return value

The function returns a String object which is the concatenation of the values occurring in the passed field. The order in which the values occur is preserved. The function returns the String that contains all concatenated values delimited by the delimiter value from the field. The order in which the values occur is preserved.

Example 1

ExampleResult
records { "x": 1 }, { "x": 2 }
aggr joinX=join(x)
{joinX: "1,2"}
records { "x": 1 }, { "x": null }
aggr joinX=join(x)
{joinX: "1"}
records { "x": 1 }, { "x": 2 }
aggr joinX=join(x, delimiter=":")
{joinX: "1:2"}
records { "x": 1 }, { "x": 2 }
aggr joinX=join(x, limit=1)
{joinX: "1"}
records { "x": 1 }, { "x": 1 }
aggr joinX=join(x, distinct=true)
{joinX: "1"}
records { "x": 1 }, { "x": 1}, { "x": 2 }, { "x": 3 }
aggr joinX=join(x, distinct=true limit=2)
{joinX: "1,2"}

Example 2

Join values form the age field for each person. The field has the same value of 56 in all objects.

src stream="testdata" | aggr fJoin = join(age) by fName, lName unwind=true | limit 1

Result:

[
{
"fJoin": "56,56,56,56,56,56,56,56,56,56"
}
]


dc

The function returns the number of unique values of a field.

Syntax

dc(field [, delimiter=string distinct=boolean limit=integer]) : number

Parameters

NameRequired, TypeDescription
fieldyesName of the field which values will be the aggregated.
normalSetPrecisionnoRange 0-sparseSetPrecision. Default 14.
sparseSetPrecisionnoRange 0-32; default is 25.

Return value

The function returns the number of unique values in the field field. The returned value is an estimated result of HyperLogLog++ algorithm.

Example 1

The task is to calculate a number of unique values of the age field for each person. The field contains the same value for every object belonging to a person, so the result is 1.

src stream="testdata" | aggr fDc = dc(age) by fName, lName unwind=true | limit 1

Result:

[
{
"fDc": 1
}
]

Example 2

The task is to calculate a number of unique values of the balance field for each person. The field contains different values for every object belonging to a person, so the result is a number of objects for each person, 10.

src stream="testdata" | aggr fDc = dc(age) by fName, lName unwind=true | limit 1

Result:

[
{
"fDc": 10
}
]


prc

The function returns k-th value of 100-quantile (percentile) for each group defined by the "BY" clause of the "AGGR" command.

Syntax

prc(field [, k=double]) : number

Parameters

NameRequired, TypeDescription
fieldYesThe field name used by the accumulative command.
kNo; doublek-th percentile value from range k in <0, 100>. The default value is 95.0.

Return value

The result is usually a floating-point number. To calculate a percentile value, the T Digest algorithm is used.

Example 1

ExampleResult
records { "x": 1 }, { "x": 2 }, { "x": 1 }, { "x": 4 }, { "x": 6 }, { "x": 11 }
aggr percentile50=prc(x, percentile=50.0)
{percentile50: 4 }
records { "x": 1 }, { "x": 2 }, { "x": 1 }, { "x": 4 }, { "x": 6 }, { "x": 11 }
aggr percentileDefault=prc(x)
{percentileDefault: 11 }
records { "x": 1 }, { "x": 2 }, { "x": 1 }, { "x": 4 }, { "x": 6 }, { "x": 11 }
aggr percentile10=prc(x, percentile=10.0)
{percentile10: 1 }

prc95

The function returns the value of 95 of 100-quantile (percentile) for each group defined by the "BY" clause of the "AGGR" command.

Syntax

prc95(field) : number

Parameters

NameRequired, TypeDescription
fieldyesThe field name used by the accumulative command.

Return value

The result is usually a floating-point number. The function returns 95 percentile value. To calculate the result value, the T Digest algorithm is used.

Example 1

ExampleResult
records { "x": 1 }, { "x": 2 }, { "x": 1 }, { "x": 4 }, { "x": 6 }, { "x": 11 }
aggr percentile95=prc95(x)
result {percentile95: 11 }
records { "x": 1 }, { "x": 1}, { "x": 1 }, { "x": 4 }, { "x": 1 }, { "x": 1 }
aggr percentile95=prc95(x)
result {percentile95: 4}